Support custom CRDs for ExtensionServer in Standalone Mode #7331

daum3ns · 2025-10-24T14:23:14Z

This change enables the file-based (standalone) provider to understand and load custom resources that are defined by an extension server.

the LoadResourcesFromYAMLBytes now accepts a server config pointer. When a resource kind is unknown, the loader checks the configured ExtensionManager and, if the GVK matches a declared policy resource, stores the object as an ExtensionServerPolicy
- Updated callers to pass the new parameter
the newOfflineGatewayAPIClient function now accepts extensionServerPoliciesGVKs, which are then registered in the scheme. this allows the offline reconciler to manipulate custom resources
added testcases for decoder, offline_controller and config_loader,
- the config reload test starts Envoy Gateway in standalone mode, hot-reloads a configuration that adds an ExtensionManager with policy resources, and verifies the new GVKs are recognized

Fixes #7141

Release Notes: Yes/No

Notes:
I found out that we most probably have the same problem with ExtensionManagers backendResources and resources field (at least the config_loader test can easily be extended to show that the registration in the scheme is missing).

I think it could be fixed in a similar way (although not sure whether the translator needs to be adapted as well). See the commented code snippets...

However, I'm very new to envoy-gateway so I'm still raising this PR to get a first, much appreciated feedback whether it makes sense to continue in this direction..

codecov · 2025-10-28T21:39:40Z

Codecov Report

❌ Patch coverage is 89.41176% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.41%. Comparing base (e781fbd) to head (676c888).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/provider/file/store.go	25.00%	4 Missing and 2 partials ⚠️
internal/gatewayapi/runner/runner.go	0.00%	2 Missing ⚠️
internal/provider/file/file.go	90.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7331      +/-   ##
==========================================
+ Coverage   72.26%   72.41%   +0.14%     
==========================================
  Files         231      231              
  Lines       34071    34128      +57     
==========================================
+ Hits        24622    24714      +92     
+ Misses       7674     7646      -28     
+ Partials     1775     1768       -7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

daum3ns · 2025-10-30T12:15:56Z

@shawnh2 i have added a test scenario which shows we have the same problem for the other 2 resources fields in the extension manager in standalone mode....
f996f5a

i think i will address this in this PR as well.. (or would you prefer a separate Issue/PR ?)
Can you answer this question here?
https://github.com/daum3ns/envoy-gateway/blob/support-custom-CRDs-for-ExtensionServer-in-Standalone-Mode/internal/gatewayapi/resource/load.go#L454
Im not sure whether all extensions should go in the same field or wether there are dedicated fields for the different types...

daum3ns · 2025-11-04T14:24:29Z

I figured it out, extensionRefFilter is the correct place to put it, also had to persist it ..
see ed62242 and 63e49df

internal/gatewayapi/resource/load.go

internal/provider/kubernetes/controller_offline_test.go

shawnh2

LGTM! Thanks

arkodg · 2025-11-06T02:43:03Z

internal/gatewayapi/resource/load.go

 				Spec: typedSpec.(gwapiv1b1.ReferenceGrantSpec),
 			}
 			resources.ReferenceGrants = append(resources.ReferenceGrants, referenceGrant)
+		default:


can we improve this logic and avoid labels

one recommendation is to setup a gvkMap in the beginning of the func, populate it based off EnvoyGateway and use it here

yes this is a good idea!

arkodg · 2025-11-06T02:44:03Z

internal/gatewayapi/runner/runner.go

 						extGKs = append(extGKs, schema.GroupKind{Group: gvk.Group, Kind: gvk.Kind})
 					}
+					// Include policy resources so extension-defined custom policies are recognised.
+					for _, gvk := range r.EnvoyGateway.ExtensionManager.PolicyResources {


is this a bug fix @guydc

daum3ns · 2025-11-06T10:43:41Z

i think i found yet another problem in standalone mode :D
when comparing the WithIndex calls in the creation of the controller_offline.go to internal/provider/kubernetes/indexers.go it looks like these indices are missing:

clusterTrustBundleBackendIndex
clusterTrustBundleBtlsIndex
clusterTrustBundleCtpIndex
secretBackendIndex
configMapBackendIndex

is this intentional or just got forgotten?
i will try to write a test for this...

daum3ns · 2025-11-07T09:52:33Z

the test shows the problem ...
the first assertion is okay but the second fails:

Error:      	Received unexpected error:
       List on GroupVersionKind gateway.envoyproxy.io/v1alpha1, Kind=Backend specifies selector on field secretBackendIndex, but no index with name secretBackendIndex has been registered for GroupVersionKind gateway.envoyproxy.io/v1alpha1, Kind=Backend
Test:       	TestNewOfflineGatewayAPIControllerIndexRegistration/offline_controller_creation_list_index_test

when I register the tested index in the offline_controller, it succeeds...

return fake.NewClientBuilder().
		WithScheme(scheme).
[...]
WithIndex(&egv1a1.Backend{}, secretBackendIndex, secretBackendIndexFunc).
[...]

shawnh2 · 2025-11-08T02:52:48Z

good catch @daum3ns , those missing indexes were introduced by #6353,feel free to make offline controller indexer complete.

arkodg · 2025-11-08T14:41:55Z

internal/gatewayapi/resource/load.go

+				un.SetNamespace(namespace)
+				switch category {
+				case extFilter, extBackend:
+					resources.ExtensionRefFilters = append(resources.ExtensionRefFilters, *un)


is this logic right of adding the ext backend into ext ref flters ?
cc @Xunzhuo

i think so, the controller does it the same way
#7331 (comment)

daum3ns · 2025-11-11T08:36:39Z

good catch @daum3ns , those missing indexes were introduced by #6353,feel free to make offline controller indexer complete.

i have added all missing indices, however the test could be improved...

daum3ns · 2025-11-11T09:33:17Z

strange.. the failing conformance test succeeds locally on my machine...

…nt scheme Teach the offline controller to recognise custom extension-server policy GVKs supplied by an extension so they can be decoded and later processed by the translator. - newOfflineGatewayAPIClient now accepts extServerPoliciesGVKs - each GVK is registered as Unstructured/UnstructuredList in the fake client's scheme - call-site updated to pass the collected policy GVKs Signed-off-by: daum3ns <[email protected]>

…backend resource registration Signed-off-by: daum3ns <[email protected]>

when reading from yaml Signed-off-by: daum3ns <[email protected]>

Add policy resources from ExtensionManager to extGKs so custom extension policies are recognised during translation. Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

server and policy resource Signed-off-by: daum3ns <[email protected]>

…f an EnvoyGateway configuration containing a standalone extension server. Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

yaml and it gets the default namespace Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

can check custom resource definitions Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

ExtensionManager.resources and ExtensionManager.backendResources, use assertions Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

… ExtensionRefFilters Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

extensionmanager.backendResources Signed-off-by: daum3ns <[email protected]>

instead of srv.EnvoyGateway (replace function parameter, adapt callers and tests) Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

and to avoid the loop over every resource in extensionanager. Signed-off-by: daum3ns <[email protected]>

unregistered list index Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

registered indices Signed-off-by: daum3ns <[email protected]>

This reverts commit 2e48e33. Signed-off-by: daum3ns <[email protected]>

Signed-off-by: daum3ns <[email protected]>

daum3ns · 2025-11-12T10:43:50Z

i cant resolve this locally.. the test succeeds....

daum3ns requested a review from a team as a code owner October 24, 2025 14:23

arkodg requested a review from shawnh2 October 24, 2025 14:25

arkodg added this to the v1.6.0 Milestone milestone Oct 24, 2025

daum3ns force-pushed the support-custom-CRDs-for-ExtensionServer-in-Standalone-Mode branch from 1f7a84b to d66478e Compare October 24, 2025 14:29

shawnh2 reviewed Nov 5, 2025

View reviewed changes

shawnh2 previously approved these changes Nov 6, 2025

View reviewed changes

shawnh2 requested review from a team November 6, 2025 00:56

arkodg reviewed Nov 6, 2025

View reviewed changes

daum3ns dismissed shawnh2’s stale review via 568f6fc November 6, 2025 10:09

arkodg reviewed Nov 8, 2025

View reviewed changes

arkodg modified the milestones: v1.6.0 Milestone, Backlog Nov 9, 2025

daum3ns force-pushed the support-custom-CRDs-for-ExtensionServer-in-Standalone-Mode branch from 6b145ed to 19fe896 Compare November 11, 2025 12:35

zhaohuabing force-pushed the support-custom-CRDs-for-ExtensionServer-in-Standalone-Mode branch from 3825255 to 7c44f70 Compare November 11, 2025 12:52

daum3ns added 6 commits November 11, 2025 23:48

controller_offline.go: add comments about possible missing extension …

b5a11cd

…backend resource registration Signed-off-by: daum3ns <[email protected]>

load.go: temporary treat all unknown kinds as extension server policy

52f87ac

when reading from yaml Signed-off-by: daum3ns <[email protected]>

runner.go: pass extension policy GVKs in translator

2656f7f

Add policy resources from ExtensionManager to extGKs so custom extension policies are recognised during translation. Signed-off-by: daum3ns <[email protected]>

file provider: persist ExtensionServerPolicies to store

8be0086

Signed-off-by: daum3ns <[email protected]>

test: add offline controller test for custom extension resources

e81baf1

Signed-off-by: daum3ns <[email protected]>

daum3ns added 24 commits November 11, 2025 23:48

config loader test: add testcase for standalone mode with extension

3176745

server and policy resource Signed-off-by: daum3ns <[email protected]>

test(config/decoder): Add a new test case to that verifies decoding o…

6beec4b

…f an EnvoyGateway configuration containing a standalone extension server. Signed-off-by: daum3ns <[email protected]>

set default namespace in custom resources if no namespace is provided

271224a

Signed-off-by: daum3ns <[email protected]>

test/controller_offline: extend test to verify the CR can be loaded from

acc84fa

yaml and it gets the default namespace Signed-off-by: daum3ns <[email protected]>

add missin yaml for test

310229c

Signed-off-by: daum3ns <[email protected]>

configloader test: fix race condition

c7990fe

Signed-off-by: daum3ns <[email protected]>

add configuration to provider struct, pass it along so that the loader

cc48acd

can check custom resource definitions Signed-off-by: daum3ns <[email protected]>

add missing yaml for decoder test

73d2f3f

Signed-off-by: daum3ns <[email protected]>

fix linter warnings

f9076f0

Signed-off-by: daum3ns <[email protected]>

controller_offline_test: extend test to show the issue also affects

85d850e

ExtensionManager.resources and ExtensionManager.backendResources, use assertions Signed-off-by: daum3ns <[email protected]>

fix testcase, register all extensions im scheme

b34877d

Signed-off-by: daum3ns <[email protected]>

load.go: store route filter resources and custom backend resources in…

0bea998

… ExtensionRefFilters Signed-off-by: daum3ns <[email protected]>

file provider: persist ExtensionRefFilters to store

d16333f

Signed-off-by: daum3ns <[email protected]>

controller_offline_test: adapt test expectation

6cca830

Signed-off-by: daum3ns <[email protected]>

typo

73603d1

Signed-off-by: daum3ns <[email protected]>

adapt YAML file for controller_offline.test

bfc83c6

Signed-off-by: daum3ns <[email protected]>

extend decoder test with extensionmanager.resources and

b87e5d8

extensionmanager.backendResources Signed-off-by: daum3ns <[email protected]>

revises from review: rename loop variable, use envoyGateway directly

73d0323

instead of srv.EnvoyGateway (replace function parameter, adapt callers and tests) Signed-off-by: daum3ns <[email protected]>

fix linter errors

83e3b28

Signed-off-by: daum3ns <[email protected]>

Refactor loadKubernetesYAMLToResources to get rid of the default label

a0fb40d

and to avoid the loop over every resource in extensionanager. Signed-off-by: daum3ns <[email protected]>

controller_offline_test: add testcase that shows problem with

361e745

unregistered list index Signed-off-by: daum3ns <[email protected]>

controller_offline: register missing indices

d2847aa

Signed-off-by: daum3ns <[email protected]>

update helm module to fix vulnerability in containerd

2e48e33

Signed-off-by: daum3ns <[email protected]>

controller_offline_test: extend index registration test to check all

b426305

registered indices Signed-off-by: daum3ns <[email protected]>

zhaohuabing force-pushed the support-custom-CRDs-for-ExtensionServer-in-Standalone-Mode branch from 7c44f70 to b426305 Compare November 11, 2025 15:48

daum3ns added 2 commits November 11, 2025 17:17

Revert "update helm module to fix vulnerability in containerd"

a7478c4

This reverts commit 2e48e33. Signed-off-by: daum3ns <[email protected]>

linter

676c888

Signed-off-by: daum3ns <[email protected]>

Support custom CRDs for ExtensionServer in Standalone Mode #7331

Are you sure you want to change the base?

Support custom CRDs for ExtensionServer in Standalone Mode #7331

Uh oh!

Conversation

daum3ns commented Oct 24, 2025

Uh oh!

codecov bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

daum3ns commented Oct 30, 2025

Uh oh!

daum3ns commented Nov 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shawnh2 left a comment

Choose a reason for hiding this comment

Uh oh!

arkodg Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

daum3ns Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

arkodg Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daum3ns commented Nov 6, 2025

Uh oh!

daum3ns commented Nov 7, 2025

Uh oh!

shawnh2 commented Nov 8, 2025

Uh oh!

arkodg Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

daum3ns Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

daum3ns commented Nov 11, 2025

Uh oh!

daum3ns commented Nov 11, 2025

Uh oh!

daum3ns commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Oct 28, 2025 •

edited

Loading

arkodg Nov 6, 2025 •

edited

Loading